Electronic device and operating method thereof

ABSTRACT

Disclosed is an electronic device that includes: a camera; a display; a memory storing one or more instructions; and a processor configured to execute the one or more instructions to obtain, by the camera, an image including at least one object, identify an object type of the at least one object, obtain spatial information corresponding to the object type, based on the object type, and control the display, to display, a virtual object, based on the spatial information.

TECHNICAL FIELD

The disclosure relates to the field of augmented reality, and inparticular to an electronic device for displaying a virtual image byusing a three-dimensional (3D) image of an object included in atwo-dimensional (2D) image, and a method of operating the electronicdevice.

BACKGROUND ART

Augmented reality (AR) is technology that displays an image byprojecting a virtual image onto a physical environment space of the realworld or a real world object. An AR device enables a user wearing the ARdevice on his/her face or head to view a real scene and a virtual imagetogether through a display module positioned in a see-through form infront of the user's eyes.

For the AR device to provide various services using a real world object,such as naturally displaying a virtual image on a real scene, the ARdevice needs to obtain required information about a real world spaceaccording to an AR service that will be provided to a user.

Accordingly, studies into a method of obtaining information about a realworld space with a small amount of computation and providing an ARservice at high speed are needed.

DESCRIPTION OF EMBODIMENTS Technical Problem

Provided are an electronic device for displaying a virtual object byusing a two-dimensional (2D) image obtained through a camera module, anda method of operating the electronic device.

Provided are an electronic device for displaying a virtual object byusing spatial information corresponding to an object included in a 2Dimage, and a method of operating the electronic device.

Technical Solution to Problem

Embodiments of the disclosure overcome the above disadvantages and otherdisadvantages not described above. Also, the disclosure is not requiredto overcome the disadvantages described above, and an embodiment of thedisclosure may not overcome any of the problems described above.Additional aspects of the disclosure will be set forth in part in thedescription which follows and additional aspects will be apparent fromthe description, or may be learned by practice of the disclosure.

In accordance with an aspect of the disclosure, an electronic deviceincludes: a camera a display; a memory storing one or more instructions;and a processor configured to execute the one or more instructions toobtain an image comprising at least one object by the camera, identifyan object type of the at least one object, obtain a spatial informationcorresponding to the object type, based on the object type, and controlthe display to display, a virtual object through the display, based onthe spatial information.

In accordance with an aspect of the disclosure, a method of operating anelectronic device includes: obtaining an image including at least oneobject through a camera identifying an object type of the at least oneobject; obtaining a spatial information corresponding to the objecttype, based on the object type; and controlling a display, to display avirtual object, based on the spatial information.

In accordance with an aspect of the disclosure, a computer-readablerecording medium having recorded thereon a program for executing theabove-described method on a computer may be provided.

BRIEF DESCRIPTION OF DRAWINGS

The above and other aspects and features of certain embodiments of thepresent disclosure will become apparent from the following descriptiontaken in conjunction with the accompanying drawings in which:

FIG. 1 is a view for schematically describing an embodiment;

FIG. 2 is a view for describing operations of an electronic deviceaccording to an embodiment, and an external server;

FIG. 3 is a block diagram of an electronic device according to anembodiment;

FIG. 4 is a flowchart for describing operations of an electronic deviceaccording to an embodiment;

FIG. 5 is a flowchart for describing a method of obtaining spatialinformation from an external server in an electronic device, accordingto an embodiment;

FIG. 6 is a view for describing an example of obtaining an imageincluding at least one object in an electronic device, according to anembodiment;

FIG. 7 is a view for describing an example of spatial informationcorresponding to an object type, according to an embodiment;

FIG. 8 is a flowchart for describing a method of generatingthree-dimensional (3D) shape information, according to an embodiment;

FIG. 9A is an exemplary view for describing a method of generating 3Dshape information, according to an embodiment;

FIG. 9B is an exemplary view for describing a method of generating 3Dshape information, according to an embodiment;

FIG. 9C is an exemplary view for describing a method of generating 3Dshape information, according to an embodiment;

FIG. 10 is a view for describing an example of 3D shape informationaccording to an embodiment;

FIG. 11 is a view for describing an example of displaying a virtualobject, according to an embodiment; and

FIG. 12 is a block diagram of an electronic device according to anembodiment, and an external server.

MODE OF DISCLOSURE

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Embodiments will be described with reference to the accompanyingdrawings. However, the disclosure is not limited to the embodimentsdescribed below, but may be implemented in several forms and may bevariously modified. A description for these embodiments is provided onlyto make the disclosure complete and allow those skilled in the art towhich the disclosure pertains to completely recognize the scope of theembodiments. In the accompanying drawings, sizes of components may beenlarged as compared with actual sizes for convenience of explanation,and ratios of the respective components may be exaggerated or reduced.

Although general terms being widely used were selected as terminologyused in the disclosure while considering the functions of thedisclosure, they may vary according to intentions of one of ordinaryskill in the art, judicial precedents, the advent of new technologies,and the like. Hence, the terms used in the disclosure must be definedbased on the meanings of the terms and the contents of the entirespecification, not by simply stating the terms themselves.

Also, it will be understood that, although the terms first, second, etc.may be used herein to describe various components, these componentsshould not be limited by these terms. These terms are only used todistinguish one component from another.

Also, the terms used in the disclosure are used to describe specificembodiments, not for the purpose of limiting the disclosure. It is to beunderstood that the singular forms “a,” “an,” and “the” include pluralreferents unless the context clearly dictates otherwise. Also, in thisspecification, it will be understood that the case in which a certainpart is “connected” to another part includes the case in which the partis “electrically connected” to the other part with another device inbetween, as well as the case in which the part is “directly connected”to the other part. Also, it will be understood that when a certain part“includes” a certain component, the part does not exclude anothercomponent but can further include another component, unless the contextclearly dictates otherwise.

The term “said” and the similar terms used in the present specification,specifically, in the claims may indicate both single and plural. Also,when the order of operations for describing a method according to thedisclosure is not definitely specified, the operations may be performedin appropriate order. However, the disclosure is not limited to theorder in which the operations are described.

The phrases “in some embodiments” or “according to an embodiment”appearing in the present specification do not necessarily indicate thesame embodiment.

Some embodiments of the disclosure may be represented by functionalblock configurations and various processing operations. The entire or apart of the functional blocks may be implemented with various numbers ofhardware and/or software configurations to execute specific functions.For example, the functional blocks of the disclosure may be implementedwith one or more microprocessors, or with circuit configurations forpredetermined functions. Also, for example, the functional blocks of thedisclosure may be implemented with various programming or scriptinglanguages. The functional blocks may be implemented with algorithms thatare executed by one or more processors. Also, the disclosure may adopttypical technologies for electronic environment settings, signalprocessing, and/or data processing. The terms “mechanism”, “element”,“means”, and “configuration” can be broadly used, and are not limited tomechanical and physical configurations.

Also, connection lines or connection members between components shown inthe drawings are examples of functional connections and/or physical orcircuital connections. In an actual apparatus, the connections betweenthe components may be implemented in the form of various functionalconnections, physical connections, or circuital connections that can bereplaced or added.

In the disclosure, ‘augmented reality (AR)’ means showing a virtualimage in a physical environment space of a real world or showing a realworld object and a virtual object together.

Also, an ‘AR device’ may be a device capable of representing ‘AR’, andmay include, as well as AR glasses being in the form of glasses, which auser wears generally on his/her face, a head mounted display (HMD)apparatus that is mounted on a head, an AR helmet, etc.

Meanwhile, a ‘real scene’ may be a real world scene that a user viewsthrough an AR device, and may include a real world object. Also, a‘virtual image’ may be an image generated through an optical engine, andinclude both a static image and a dynamic image. A virtual image may beshown together with a real scene, and may be an image representinginformation about a real world object in a real scene, information aboutan operation of an AR device, a control menu, etc.

Accordingly, a general AR device may include an optical engine forgenerating a virtual image configured with light generated from a lightsource, and a waveguide for guiding a virtual image generated by theoptical engine to a user's eyes, the waveguide formed with a transparentmaterial to enable the user to view the virtual image together with areal world scene. As described above, because the AR device needs toshow a real world scene together with a virtual image, an opticalelement for changing a path of light basically having straightness maybe needed to guide light generated by the optical engine to a user'seyes through the waveguide. The path of light may change throughreflection by a mirror, etc. or through diffraction by a diffractiveelement, such as a diffractive optical element (DOE), a holographicoptical element (HOE), etc., although not limited thereto.

Moreover, terms described in the specification such as “part,” “module,”and “unit,” refer to a unit of processing at least one function oroperation, and may be implemented by software, a hardware component suchas a field-programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC), or a combination of software and hardware.

Reference numerals used for method steps are just used for convenienceof explanation, but not to limit an order of the steps. Thus, unless thecontext clearly dictates otherwise, the written order may be practicedotherwise.

Hereinafter, embodiments according to the disclosure will be describedin detail with reference to the accompanying drawings.

FIG. 1 is a view for schematically describing an embodiment.

The electronic device 100 according to an embodiment may be an augmentedreality (AR) device. For example, as shown in FIG. 1 , the electronicdevice 100 may be implemented in the form of glasses that a user maywear on his/her face. Also, the electronic device 100 may be implementedin the form of goggles, a helmet, a hat, etc. which a user mounts onhis/her head, although not limited thereto.

The electronic device 100 may display, when providing an AR service, avirtual object of a virtual image through a display 140 (see FIG. 3 ).For the electronic device 100 to realistically display a virtual objectaround a real world object, information about a shape, size, depth,orientation, etc. of the real world object may be needed.

As illustrated in FIG. 1 , for example, a user may wear the electronicdevice 100 being in the form of glasses on his/her face and view a realscene 110. The real scene 110 may include a real object (for example, atable) 113. The electronic device 100 may display virtual objects 115and 117 through the display 140 (see FIG. 3) such that the virtualobjects 115 and 117 are shown by the user wearing the electronic device100 as though they are positioned around the real object 113 in the realscene 110.

According to an embodiment, the electronic device 100 may performrendering such that a part of the virtual objects 115 and 117,obstructed by the real object 113 in a direction which the user views,is omitted and the other part of the virtual objects 115 and 117, notobstructed by the real object 113 in the direction which the user views,is displayed by the display 140.

Accordingly, the electronic device 100 may provide, when displaying avirtual object, an obstruction effect with respect to an overlappingpart between a real object and the virtual object. The obstructioneffect is to realistically represent a virtual object around, in frontof, or behind the real object. That is, the obstruction effect is torepresent the virtual object, when it is assumed that the virtual objectexists around real objects, as though the virtual object is positionedbetween the real objects, by displaying an area of the virtual object,located in front of the real objects, among overlapping areas betweenthe virtual object and the real objects for a user and not displayingthe other area of the virtual object, located behind the real objects.

To display a virtual object while distinguishing an area obstructed by areal object from an area not obstructed by the real object, theelectronic device 100 according to an embodiment may photograph the realscene 110 through a camera 175 (see FIG. 3 ) of the electronic device100 to obtain a two-dimensional (2D) image and use a spatial informationabout a real object included in the 2D image. The electronic device 100may identify an object type of the real object included in the 2D image,and obtain the spatial information of the real object from an externalserver by using the identified object type. The electronic device 100may display the virtual object around the real object by using thespatial information obtained from the external server.

A detailed method for displaying a virtual object by using a 2D image inthe electronic device 100 will be described with reference to thedrawings which will be described at a later time.

According to an embodiment, the electronic device 100 may provide anobstruction effect when displaying a virtual object, by using a 2D imagegenerated by unidirectional photographing, without 360 degreephotographing with a plurality of cameras or a depth sensor. Also, theelectronic device 100 may provide an AR service of providing theobstruction effect at high speed with a small amount of computation,without having to perform a process of calculating depth information ofan object, the process requiring a large amount of computation.

FIG. 2 is a view for describing operations of an electronic deviceaccording to an embodiment and an external server.

The electronic device 100 according to an embodiment may be a mobileelectronic device of a portable type, or a fixed type electronic devicepositioned at a fixed location. For example, in FIG. 2 , the electronicdevice 100 may be a wearable device 101, a smart phone 102, a tablepersonal computer (PC) 103, a laptop computer, a television (TV), adesktop computer, an e-book terminal, a digital broadcasting terminal, apersonal digital assistant (PDA), a portable multimedia player (PMP), adigital camera, a camcorder, a navigation, a media player, a microserver, a global positioning system (GPS) device, or a MP3 player,although not limited thereto.

The electronic device 100 according to an embodiment maytransmit/receive data to/from an external server 200 through acommunication network.

The communication network according to an embodiment may be configuredwith at least one of a wired communication network or a wirelesscommunication network. More specifically, the communication network maybe mobile communication (for example, wireless broadband (Wibro), worldinteroperability for microwave access (Wimax), code division multipleaccess (CDMA), wideband code division multiple access (WCDMA),3generation (3G), 4generation (4G), 5generation (5G), etc.), short-rangecommunication (for example, near field communication (NFC), Bluetooth,wireless LAN (WLAN), wireless-fidelity (Wi-Fi), etc.), and/or low-powerwide area communication (for example, TV white space (TVMS), weightless,etc.).

The electronic device 100 according to an embodiment may obtain spatialinformation corresponding to an object included in a 2D image from anexternal server 200.

The external server 200 according to an embodiment may store spatialinformation for object types. According to an embodiment, an object typemay be information representing what the object is. An object type maybe a common term used to name the same kind of the object. For example,referring to FIG. 1 , an object type of the real object 113 included inthe real scene 110 may be identified as a table.

According to an embodiment of the disclosure, the spatial informationfor object types may include information about three-dimensional (3D)shapes representing preset object types. In FIG. 1 , the spatialinformation for object types may be information about a 3D shaperepresenting the real object 113, for example, ‘table’. For example, theserver 200 may store spatial information including a 3D shaperepresenting ‘table’ in correspondence to ‘table’.

According to an embodiment, the spatial information for object types maybe stored in a form of a table including 3D shapes corresponding to theindividual object types, in a memory 230 (see FIG. 12 ) of the externalserver 200. Also, the spatial information for object types may be storedin a form of an algorithm for detecting the spatial informationcorresponding to the object types, in the memory 230 of the externalserver 200.

The electronic device 100 according to an embodiment may request theexternal server 200 to provide spatial information corresponding to anobject type, and the external server 200 may transmit the spatialinformation corresponding to the object type to the electronic device100 through a communication network in response to the request from theelectronic device 100.

FIG. 3 is a block diagram of an electronic device according to anembodiment.

According to an embodiment, the electronic device 100 may be an ARdevice having a communication function and a data processing functionand configured to provide AR images, although not limited thereto.

In FIG. 3 , the electronic device 100 according to an embodiment mayinclude a memory 130, a processor 120, a display 140, a sensor 150, acamera 175, a communicator 180, a sound outputter 185, a vibration motor187, a microphone 190, and a user inputter 195. However, all componentsillustrated in FIG. 3 may not be indispensable components of theelectronic device 100. The electronic device 100 may be implemented withmore components than those illustrated in FIG. 3 or less components thanthose illustrated in FIG. 3 .

The processor 120 of the electronic device 100 may execute programsstored in the memory 130 to control overall operations of the display140, the sensor 150, the camera 175, the communicator 180, the soundoutputter 185, the vibration motor 187, the microphone 190, the userinputter 195, etc.

The memory 130 according to an embodiment may store programs that are tobe executed by the processor 120, and data input to the electronicdevice 100 or that is to be output from the electronic device 100.

The memory 130 may include at least one type of storage medium among aflash memory type, a hard disk type, a multimedia card micro type, acard type memory (for example, SD or XD memory), random access memory(RAM), static random access memory (SRAM), read only memory (ROM),electrically erasable programmable read-only memory (EEPROM),programmable read-only memory (PROM), a magnetic memory, a magneticdisk, or an optical disk.

The programs stored in the memory 130 may be classified into a pluralityof software modules according to their functions, and for example, thesoftware modules may include an image obtaining module 131, an objectidentifying module 132, a spatial information obtaining module 133, a 3Dshape information generating module 134, and a virtual object displaymodule 135, although not limited thereto. However, the software modulesmay store some of the above-mentioned modules or may further includeanother software module.

Also, according to an embodiment, the memory 130 may store a 2D imageobtained through the camera 175. Also, the memory 130 may store spatialinformation for object types, received from the external server 200.

The processor 120 may control overall operations of the electronicdevice 100. The processor 120 may control operations or functions thatare performed by the electronic device 100 by executing instructions orprograms stored in the memory 130.

According to an embodiment, the processor 120 may be configured with asingle processor or a plurality of processors. The processor 120 mayinclude, for example, at least one hardware of a central processing unit(CPU), a microprocessor, a graphics processing unit (GPU), applicationspecific integrated circuits (ASICs), digital signal processors (DSPs),digital signal processing devices (DSPDs), programmable logic devices(PLDs), or field programmable gate arrays (FPGAs), although not limitedthereto. The processor 120 may obtain a 2D image through the camera 175by executing the image obtaining module 131 stored in the memory 130.

According to an embodiment, the processor 120 may obtain a 2D image byphotographing surroundings of the electronic device 100 through thecamera 175 at preset time intervals after the electronic device 100 ispowered on. Alternatively, the processor 120 may obtain a 2D imagethrough the camera 175, when a preset application requiring displayingof a virtual object is executed.

For example, after a user wears the electronic device 100 on his/herface, the electronic device 100 may photograph a real scene of aphotographable field of view (FOV) through the camera 175 at preset timeintervals. In this case, an image obtained through the camera 175 may bea 2D image obtained by photographing a real 3D space in one direction.

The processor 120 may identify at least one object from the 2D image byexecuting the object identifying module 132 stored in the memory 130.

The processor 120 may detect at least one object included in the 2Dimage from the 2D image by using a preset object detection algorithm.

Also, according to an embodiment, the processor 120 may identify anobject type of the detected object. The processor 120 may identify theobject type of the object included in the 2D image by using a trainedmodel trained by using an artificial intelligence (AI) algorithm toidentify an object type of an object.

The processor 120 may obtain a spatial information corresponding to theobject type of the object included in the 2D image from the externalserver 200, by executing the spatial information obtaining module 133stored in the memory 130.

The processor 120 may transmit, to the external server 200 through thecommunicator 180, a spatial information request signal includinginformation about the object type, based on the object type identifiedby the object identifying module 132. The external server 200 maydetermine spatial information corresponding to the object type receivedfrom the electronic device 100, based on the spatial information forobject types, stored in the memory 230. The external server 200 maytransmit the spatial information corresponding to the object type to theelectronic device 100, through a communicator 280.

The processor 120 may generate 3D shape information corresponding to theobject included in the 2D image by using the spatial informationcorresponding to the object type and obtained from the external server200, by executing the 3D shape information generating module 134 storedin the memory 130.

According to an embodiment of the disclosure, the 3D shape informationmay be information about a 3D shape corresponding to a size andorientation of a real object included in the 2D image obtained by thecamera 175.

According to an embodiment of the disclosure, the electronic device 100may generate 3D shape information corresponding to the size andorientation of the real object included in the 2D image, based on thespatial information corresponding to the object type obtained from theexternal server 200.

The electronic device 100 may generate 3D shape informationcorresponding to the real object by adjusting a size and orientation ofa 3D shape corresponding to the object type included in the spatialinformation obtained from the external server 200 such that the size andorientation of the 3D shape match with the size and orientation of thereal object included in the 2D image approximately within a presetrange.

The processor 120 may display a virtual object by using the 3D shapeinformation, by executing the virtual object display module 135 storedin the memory 130.

According to an embodiment, the processor 120 may control the display140 to display the virtual object, such that the virtual object looks asthough it exists around the real object, by using the 3D shapeinformation corresponding to the real object.

The processor 120 may control, when positioning the virtual objectaround the real object, the virtual object such that the virtual objectis more realistically displayed between real objects, by not renderingan area of the virtual object obstructed by the real object amongoverlapping areas between the virtual object and the real object throughthe display 140 (see FIG. 3 ).

The display 140 may output information processed by the processor 120.For example, the display 140 may display the virtual object.

According to an embodiment, the display 140 may provide an AR image. Thedisplay 140 according to an embodiment of the disclosure may include awaveguide and a display module. The waveguide may be made of atransparent material such that a user wearing the electronic device 100views outside through an area of the rear surface of the electronicdevice 100. The waveguide may be configured with a flat plate of asingle-layer or multi-layer structure made of a transparent material toenable light to be reflected and propagate thereinside. The waveguidemay be opposite to an exit surface of the display module to receivelight of a projected virtual image. Herein, the transparent materialmeans a material transmitting light. Transparency of the transparentmaterial may be not 100%, and the transparent material may have a presetcolor.

According to an embodiment of the disclosure, because the waveguide ismade of a transparent material, a user may view an outside real scenetogether with a virtual object of a virtual image, through the display140. Therefore, the waveguide may be also called a see through display.The display 140 may output a virtual object of a virtual image throughthe waveguide, thereby providing an AR image.

The sensor 150 may include a motion sensor 155 and an eye trackingsensor 157.

The motion sensor 155 may be an inertial measurement unit (IMU). The IMUmay be a combination of sensors configured to sense a motion of anobject in a three-dimensional (3D) space, that is, changes in locationand orientation. For example, the combination of sensors may include atleast one of an accelerometer, a gyroscope, or a magnetometer.

Also, the motion sensor 155 may include at least one of an accelerationsensor, a magnetic sensor, or a gyroscope sensor.

The eye tracking sensor 157 may detect gaze information of a user's eye.According to an embodiment of the disclosure, the gaze information mayinclude at least one of a gaze direction of the user's eye or a pupilposition or pupil center coordinates of the user's eye.

The eye tracking sensor 157 may provide light to a user's eye (left orright eye), and sense an amount of light reflected from the user's eye.The eye tracking sensor 157 may detect a gaze direction, a pupilposition, pupil center coordinates, etc. of the user's eye, based on thesensed amount of light.

Also, the eye tracking sensor 157 may provide light to the user's eye,and photograph the user's eye to generate an image of the user's eye.The eye tracking sensor 157 may detect a gaze direction, a pupilposition, pupil center coordinates, etc. of the user's eye, based on theimage of the user's eye.

The camera 175 may photograph surroundings of the electronic device 100.The camera 175 may obtain an image frame, such as a still image, amoving image, etc., through an image sensor, when an applicationrequiring a photographing function is executed.

An image captured through the image sensor may be processed through theprocessor 120 or a separate image processor (not shown). Also, thecaptured image may be displayed on the display 140.

Also, an image frame processed by the processor 120 or the separateimage processor (not shown) may be stored in the memory 130 ortransmitted to outside through the communicator 180. Two or more camera175 may be provided according to a configuration aspect of theelectronic device 100.

According to an embodiment, the camera 175 may obtain a 2D image byphotographing a real scene of surroundings of the electronic device 100at preset time intervals. Also, the camera 175 may photograph, when anapplication requiring displaying of a virtual image is executed, a realscene of surroundings of the electronic device 100 to obtain a 2D image.

The communicator 180 may include at least one component for enablingcommunications between the electronic device 100 and the external server200 or an external device (not shown).

For example, the communicator 180 may include a short-range communicatoror a mobile communicator.

The short-range communicator may include a Bluetooth communicator, ashort-range wireless communicator (NFC/RFID), a WLAN (Wi-Fi)communicator, a Zigbee communicator, an infrared data association (IrDA)communicator, a ultra wideband (UWB) communicator, an Ant+ communicator,etc., although not limited thereto.

The mobile communicator may transmit/receive a wireless signal to/fromat least one of a base station, an external terminal, or a server on amobile communication network. Herein, the wireless signal may include avoice call signal, a video call signal or various formats of dataaccording to transmission/reception of text/multimedia messages.

According to an embodiment, the electronic device 100 may transmit aspatial information requesting signal including information about anobject type to the external server 200, through the communicator 180.The electronic device 100 may receive spatial information correspondingto the object type from the external server 200, through thecommunicator 180.

The sound outputter 185 may output audio data received from thecommunicator 180 or stored in the memory 130. Also, the sound outputter185 may output a sound signal related to a function (for example, callsignal ringtone, message ringtone, and notification sound) that isperformed in the electronic device 100.

The sound outputter 185 according to an embodiment may include aspeaker, a buzzer, etc. The sound outputter 185 according to anembodiment of the disclosure may be implemented in a form of earphonesthat are installed in or detachably attached to the electronic device100. Also, the sound outputter 185 according to an embodiment of thedisclosure may output sound by bone conduction.

The vibration motor 187 may output a vibration signal. For example, thevibration motor 187 may output vibration signals corresponding tooutputs of audio data or video data (for example, call signal ringtone,message ringtone, etc.). Also, the vibration motor 187 may output avibration signal when a user input is received by the user inputter 195.Also, the vibration motor 187 may provide notification as vibrationswhen the electronic device 100 operates in a vibration mode.

The microphone 190 may receive an external sound signal and process theexternal sound signal to electrical voice data. For example, themicrophone 190 may receive a sound signal from an external device or aspeaker. Also, the microphone 190 may receive a user's voice input forcontrolling the electronic device 100. The microphone 190 may usevarious noise removal algorithms for removing noise generated in aprocess of receiving an external sound signal.

The user inputter 195 may be a device for enabling a user to input datafor controlling the electronic device 100. For example, the userinputter 195 may include at least one of a key pad, a dome switch, atouch pad (a contact capacitive type, a pressure resistive type, aninfrared beam type, a surface acoustic wave type, an integral straingauge type, a piezo effect type, etc.), a jog wheel, or a jog switch,although not limited thereto.

FIG. 4 is a flowchart for describing operations of an electronic deviceaccording to an embodiment. FIG. 5 is a flowchart for describing amethod of obtaining spatial information from an external server in anelectronic device according to an embodiment. FIG. 6 is a view fordescribing an example of obtaining an image including at least oneobject in an electronic device according to an embodiment. FIG. 7 is aview for describing an example of spatial information corresponding toan object type according to an embodiment. FIGS. 6 and 7 are viewsreferred to describe flowcharts of FIGS. 4 and 5 .

In operation S401 of FIG. 4 , the electronic device 100 may obtain animage including at least one object through the camera 175.

According to an embodiment, when a preset application requiringdisplaying of a virtual image is executed, the electronic device 100 mayobtain an image by photographing surroundings of the electronic devicethrough the camera 175. Also, the electronic device 100 may obtain animage by photographing surroundings of the electronic device 100 throughthe camera 175 at preset time intervals.

According to an embodiment, the image obtained through the camera 175may be a 2D image obtained by photographing at least one real objectincluded in a real scene.

In FIG. 6 , the electronic device 100 may obtain a 2D image 500 byphotographing a real scene of a FOV through the camera 175. For example,the 2D image 500 may include a first object 501, a second object 502,and a third object 503.

In operation S402 of FIG. 4 , the electronic device 100 may identify anobject type of the at least one object.

According to an embodiment, the electronic device 100 may detect atleast one object included in the 2D image 500 from the 2D image 500 byusing a preset object detection algorithm.

According to an embodiment, the electronic device 100 may identify anobject type of the detected object. The electronic device 100 mayidentify the object type of the object by using a trained model trainedby using an AI algorithm to identify an object type of an object.

In FIG. 7 , for example, the electronic device 100 may identify that anobject type of the first object 501 is bookshelf, an object type of thesecond object 502 is table, and an object type of the third object 503is sofa.

In operation S403 of FIG. 4 , the electronic device 100 may obtainspatial information corresponding to the object type, based on theidentified object type.

According to an embodiment, the electronic device 100 may transmit aspatial information request signal including information about theobject type to the external server 200, and receive spatial informationcorresponding to the objet type from the external server 200.

In FIG. 7 , the electronic device 100 may obtain first spatialinformation including a first 3D shape 601 corresponding to bookshelf asa first object type of the first object 501 from the external server200. Also, the electronic device 100 may obtain second spatialinformation including a second 3D shape 602 corresponding to table as asecond object type of the second object 502 from the external server200. Also, the electronic device 100 may obtain third spatialinformation including a third 3D shape 603 corresponding to sofa as athird object type of the third object 503 from the external server 200.

A method of obtaining spatial information from the external server 200will be described in more detail with reference to FIG. 5 which will bedescribed at a later time.

In operation S404 of FIG. 4 , the electronic device 100 may display avirtual object through the display 140 based on the obtained spatialinformation.

According to an embodiment, the electronic device 100 may generate 3Dshape information corresponding to a size and orientation of an objectincluded in the image obtained by the camera 175 (see FIG. 3 ), based onthe spatial information corresponding to the object type.

A method of generating 3D shape information will be described in moredetail with reference to FIGS. 8 and 9 which will be described at alater time.

FIG. 5 is a flowchart for describing a method of obtaining spatialinformation from an external server in an electronic device.

In operation S501 of FIG. 5 , the external server 200 may store spatialinformation for object types.

According to an embodiment, the spatial information for object types mayinclude information about a 3D shape representing a preset object type.

In FIG. 7 , the external server 200 may store the first spatialinformation including the first 3D shape 601 representing bookshelf, thesecond spatial information including the second 3D shape 602representing table, and the third spatial information including thethird 3D shape 603 representing sofa.

In operation S502 of FIG. 5 , the electronic device 100 may identify anobject type of at least one object included in an image obtained by thecamera 175 (see FIG. 3 ). In operation S503, the electronic device 100may transmit a spatial information request signal including informationabout the object type to the external server 200.

In FIG. 7 , the electronic device 100 may transmit a spatial informationrequest signal including information about an object type (for example,bookshelf, table, or sofa) of an object included in a 2D image to theexternal server 200, thereby requesting spatial information including a3D shape corresponding to the object type.

In operation S504 of FIG. 5 , the external server 200 may determinespatial information corresponding to the object type.

In FIG. 7 , the external server 200 may extract the first spatialinformation including the first 3D shape 601 corresponding to bookshelf,the second spatial information including the second 3D shape 602corresponding to table, or the third spatial information including thethird 3D shape 603 corresponding to sofa, from the spatial informationabout 3D shapes for object types, stored in the memory 230 (see FIG. 12) of the external server 200, based on the information about the objecttype (for example, bookshelf, table, or sofa), received from theelectronic device 100.

According to an embodiment, a 3D shape corresponding to an object typemay not be exactly identical to a shape of a real object included in animage obtained by the camera 175 (see FIG. 3 ). A 3D shape correspondingto an object type, stored in the external server 200, may be auniversal, common 3D shape of objects having the same object type.

In operation S505 of FIG. 5 , the electronic device 100 may receive thespatial information corresponding to the object type from the externalserver 200.

In FIG. 7 , the external server 200 may transmit the first spatialinformation including the first 3D shape 601 corresponding to bookshelf,the second spatial information including the second 3D shape 602corresponding to table, and the third spatial information including thethird 3D shape 603 corresponding to sofa to the electronic device 100.

In operation S506 of FIG. 5 , the electronic device 100 may display avirtual object through the display 140 (see FIG. 3 ), based on theobtained spatial information.

The electronic device 100 may generate 3D shape informationcorresponding to a size and orientation of the object included in theimage obtained by the camera 175, based on the spatial informationcorresponding to the object type. The electronic device 100 may displaya virtual object through the display 140 (see FIG. 3 ), based on the 3Dshape information.

In FIG. 7 , the electronic device 100 may adjust a size and orientationof the first 3D shape 601 corresponding to bookshelf according to thefirst object 501 included in the 2D image 500, adjust a size andorientation of the second 3D shape 602 corresponding to table accordingto the second object 502 included in the 2D image 500, and adjust a sizeand orientation of the third 3D shape 603 corresponding to sofaaccording to the third object 503 included in the 2D image 500.

A method of generating 3D shape information will be described in moredetail with reference to FIGS. 8 and 9 which will be described below.

FIG. 8 is a flowchart for describing a method of generating 3D shapeinformation according to an embodiment. FIGS. 9A, 9B, and 9C areexemplary views for describing a method of generating 3D shapeinformation according to an embodiment. FIG. 10 is a view for describingan example of 3D shape information according to an embodiment. FIG. 11is a view for describing an example of displaying a virtual objectaccording to an embodiment. FIGS. 9A, 9B, and 9C, 10, and 11 are viewsreferred to describe the flowchart of FIG. 8 .

In operation S801 of FIG. 8 , the electronic device 100 may obtainspatial information corresponding to an object type from the externalserver 200.

In FIGS. 9A, 9B, and 9C, for example, the electronic device 100 mayobtain the first 3D shape 601 corresponding to bookshelf, the second 3Dshape 602 corresponding to table, and the third 3D shape 603corresponding to sofa, from spatial information for object types, storedin the memory 230 (see FIG. 12 ) of the external server 200, based oninformation about object types (for example, information aboutbookshelf, table, and sofa).

In operation S802 of FIG. 8 , the electronic device 100 may generate 3Dshape information corresponding to a size and orientation of an objectincluded in an image, based on the spatial information corresponding tothe object type.

In FIG. 9A, the electronic device 100 may adjust a size and orientationof a first 3D shape 601 obtained from the external server 200 accordingto a first object 501 included in a 2D image. The electronic device 100may zoom in or out the size of the first 3D shape 601 such that the sizeof the first 3D image 601 is approximately identical to a size of thefirst object 501 within a preset range. Also, the electronic device 100may determine the orientation of the first 3D shape 601 by rotating thefirst 3D shape 601 with respect to a preset central axis such that theorientation of the first 3D shape 601 is approximately identical toorientation of the first object 501 included in the 2D image within apreset range. Accordingly, the electronic device 100 may generate first3D shape information including a 3D shape 604 of which a size andorientation are approximately identical to those of the first object 501included in the 2D image within the preset ranges.

In FIG. 9B, the electronic device 100 may adjust a size and orientationof a second 3D shape 602, obtained from the external server 200,according to a second object 502 included in the 2D image. Theelectronic device 100 may zoom in or out the size of the second 3D shape602 such that the size of the second 3D shape 602 is approximatelyidentical to that of the second object 502 within a preset range. Also,the electronic device 100 may determine the orientation of the second 3Dshape 602 by rotating the second 3D shape 602 with respect to a presetcentral axis such that the orientation of the second 3D shape 602 isapproximately identical to that of the second object 502 included in the2D image within a preset range. Accordingly, the electronic device 100may generate second 3D shape information including a 3D shape 605 ofwhich a size and orientation are approximately identical to those of thesecond object 502 included in the 2D image within the preset ranges.

In FIG. 9C, the electronic device 100 may adjust a size and orientationof a third 3D shape 603, obtained from the external server 200,according to a third object 503 included in the 2D image. The electronicdevice 100 may zoom in or out the size of the third 3D shape 603 suchthat the size of the third 3D shape 603 is approximately identical tothat of the third object 503 within a preset range. Also, the electronicdevice 100 may determine the orientation of the third 3D shape 603 byrotating the third 3D shape 603 with respect to a preset central axissuch that the orientation of the third 3D shape 603 is approximatelyidentical to that of the third object 503 included in the 2D imagewithin a preset range. Accordingly, the electronic device 100 maygenerate third 3D shape information including a 3D shape 606 of which asize and orientation are approximately identical to those of the thirdobject 503 included in the 2D image within the preset ranges.

According to an embodiment of the disclosure, the electronic device 100may generate 3D shape information including the 3D shapes 604, 605, and606 respectively corresponding to the plurality of first, second, andthird objects 501, 502, and 503.

In FIG. 10 , the electronic device 100 may store a combination of the 3Dshape information including the 3D shapes 604, 605, and 606 respectivelycorresponding to the plurality of first, second, and third objects 501,502, and 503 included in the 2D image 500, in the memory 130 (see FIG. 3).

According to an embodiment, a 3D shape included in spatial informationcorresponding to an object type may be not exactly identical to a shapeof a real object included in an image. Accordingly, in FIG. 10 , the 3Dshapes 604, 605, and 606 included in the 3D shape information generatedbased on spatial information may be not exactly identical to realobjects included in the 2D image 500.

In operation S803 of FIG. 8 , the electronic device 100 may display avirtual object through the display 140 (see FIG. 3 ) based on the 3Dshape information.

The electronic device 100 may display a virtual object around the first,second, and third objects 501, 502, and 503, based on the combination ofthe 3D shape information respectively corresponding to the plurality offirst, second, and third objects 501, 502, and 503 included in the 2Dimage 500.

The electronic device 100 may display a virtual object through thedisplay 140 (see FIG. 3 ) such that the virtual object looks as thoughit exists around a real object included in a 2D image, by using 3D shapeinformation corresponding to the real object. The electronic device 100may display the virtual object in such a way to display an area of thevirtual object, located in front of the real object and not obstructedby the real object, through the display 140 (see FIG. 3 ), withoutdisplaying an area of the virtual object, obstructed by the real object.

In FIG. 11 , the electronic device 100 may display virtual objects 1001and 1002 each having a chair shape around the second object 502 (forexample, a table) which is a real object included in the 2D image 500.

The electronic device 100 may display the virtual objects 1101 and 1102such that the virtual objects 1101 and 1102 (chair shape) look as thoughthey are positioned around the second object 502 (table), by using 3Dshape information including the 3D shape 605 (see FIG. 10 )corresponding to the second object 502.

For example, the electronic device 100 may display areas 1102 and 1103not obstructed by the second object 502, through the display 140,without displaying areas 1101 and 1104 obstructed by the second object502, among overlapping areas between the virtual objects 1001 and 1002and the second object 502 as a real object. Accordingly, the electronicdevice 100 may provide an obstruction effect with respect to anoverlapping area between a real object and a virtual object.

FIG. 12 is a block diagram of an electronic device according to anembodiment and an external server.

Components of the electronic device 100 illustrated in FIG. 12 maycorrespond to those of the electronic device 100 illustrated in FIG. 3 ,and therefore, descriptions thereof will be omitted.

According to an embodiment of the disclosure, the electronic device 100may transmit/receive data to/from the external server 200 through thecommunicator 180.

The external server 200 illustrated in FIG. 12 may include a processor220, the memory 230, and the communicator 280. However, the componentsshown in FIG. 12 may not be indispensable components of the externalserver 200. The external server 200 may be implemented with morecomponents than those illustrated in FIG. 12 or less components thanthose illustrated in FIG. 12 .

The processor 220 according to an embodiment may control overalloperations of the external server 200. The processor 220 may beconfigured with a single processor or a plurality of processors. Theprocessor 220 according to an embodiment of the disclosure may executeone or more programs stored in the memory 230.

The memory 230 according to an embodiment of the disclosure may storevarious data, programs, or applications for driving and controlling theexternal server 200. The programs stored in the memory 230 may includeone or more instructions. The programs (one or more instructions) orapplications stored in the memory 230 may be executed by the processor220.

The memory 230 according to an embodiment may store programs forprocessing and controlling the processor 220. The programs stored in thememory 230 may be classified into a plurality of modules according totheir functions.

The memory 230 according to an embodiment may store spatial informationincluding 3D shapes for object types in a form of a table. Also, thememory 230 may store spatial information including 3D shapes for objecttypes in a form of an algorithm for detecting 3D shapes corresponding toobject types.

The communicator 280 may include one or more components for enablingcommunications between the external server 200 and the electronic device100 or between the external server 200 and an external device (notshown).

For example, the communicator 280 may include a short-range communicatorand a mobile communicator.

The short-range communicator may include a Bluetooth communicator, ashort-range wireless communicator (NFC/RFID), a WLAN (Wi-Fi)communicator, a Zigbee communicator, an IrDA communicator, a UWBcommunicator, an Ant+ communicator, etc., although not limited thereto.

The mobile communicator may transmit/receive a wireless signal to/fromat least one of a base station, an external terminal, or a server on amobile communication network. Herein, the wireless signal may include avoice call signal, a video call signal or various formats of dataaccording to transmission/reception of text/multimedia messages.

According to an embodiment, the external server 200 may receive aspatial information request signal including information about an objecttype from the electronic device 100, through the communicator 280. Also,the external server 200 may transmit spatial information correspondingto an object type to the electronic device 100, through the communicator280.

Meanwhile, the embodiments as described above may be written as aprogram that is executable on a computer, and implemented on ageneral-purpose digital computer that operates the program using acomputer-readable recording medium. Also, a data structure used in theabove-described embodiments of the disclosure may be recorded on acomputer-readable medium through various means. Also, theabove-described embodiments may be implemented in a form of a recordingmedium including an instruction that is executable by a computer, suchas a program module that is executed by a computer. For example, themethods may be implemented as a software module or an algorithm and maybe stored on computer-readable recording medium as computer-readable andexecutable codes or program commands.

The computer-readable medium may be arbitrary recording medium that isaccessible by a computer, and may include volatile and non-volatilemedia and separable and non-separable media. Examples of thecomputer-readable recording medium may include magnetic storage media(for example, read only memory (ROM), floppy disks, hard disks, etc.)and optical recording media (for example, compact disc-read only memory(CD-ROM), or digital versatile disc (DVD)), although not limitedthereto. Also, the computer-readable medium may include computer storagemedium and communication medium.

Also, a plurality of computer-readable recording media may bedistributed to computer systems over a network, and data (for example,program instructions and codes) stored in the distributed storage mediamay be executed by at least one computer.

Specific executions described in the disclosure may be an embodiment,and do not limit the scope of the disclosure by any method. For brevityof description, descriptions of typical electronic configurations,control systems, software, and other functional aspects of such systemsmay be omitted.

The aforementioned descriptions are only for illustrative purposes, andit will be apparent that those of ordinary skill in the art may makevarious modifications thereto without changing the technical spirit andessential features of the present disclosure. Thus, it should beunderstood that the exemplary embodiments described above are merely forillustrative purposes and not for limitation purposes in all aspects.For example, each component described as a single type may beimplemented in a distributed type, and components described asdistributed may be implemented in a combined form.

The use of all examples or exemplary terms, e.g., “etc.,” in thedisclosure is for the purpose of describing the disclosure in detail andis not intended to be limited by the scope of the claims, but is notlimited thereto.

Also, unless stated to the contrary, such as “essential”, “importantly”,etc., the components described in this disclosure may not be essentialcomponents for the performance of the disclosure.

It is to be understood that the disclosure is not limited by thespecific embodiments described in the specification and that variouschanges and modifications may be made therein without departing from thespirit and scope of the present disclosure.

It is to be understood that the disclosure is not limited by thespecific embodiments described in the specification and that variouschanges and modifications may be made therein without departing from thespirit and scope of the disclosure, and substitutions are to beunderstood as being included in this disclosure. Therefore, thedisclosed embodiments should be understood in an illustrative ratherthan a restrictive sense.

The scope of the disclosure is defined by the appended claims ratherthan the detailed description of the disclosure, and all changes ormodifications derived from the meaning and scope of the claims and theirequivalents should be construed as being included within the scope ofthe disclosure.

As used herein, the terms “portion”, “module”, or “unit” refers to aunit that can perform at least one function or operation, and may beimplemented as hardware, software, or a combination of hardware andsoftware.

The “portion”, “module”, or “unit” may be stored in an addressablestorage medium, or may be implemented by a program that can be executedby a processor.

For example, the “portion”, “module”, or “unit” may be implemented bycomponents, such as software components, object-oriented softwarecomponents, class components, and task components; and processes,functions, attributes, procedures, sub-routines, segments of programcodes, drivers, firmware, microcodes, circuits, data, databases, datastructures, tables, arrays, and variables.

In this specification, the description “A may include one of a1, a2 ora3” has a broad meaning that an exemplary element that may be includedin an element A is a1, a2 or a3.

Thus, elements that may form the element A are not necessarily limitedto a1, a2, and a3. Thus, such an expression should not be exclusivelyinterpreted that the element A only includes a1, a2, and a3.

Also, such an expression means that the element A may include a1, mayinclude a2, or may include a3. The expression does not mean that theelements forming the element A are definitely selectively determinedwithin a certain group. For example, the expression should not beinterpreted that the element A necessarily includes a1, a2, or a3selected from the group consisting of a1, a2, and a3.

Although embodiments have been described for illustrative purposes,those skilled in the art will appreciate that various modifications,additions and substitutions are possible, without departing from thescope and spirit of the disclosure. These modifications should also beunderstood to fall within the scope of the disclosure.

The invention claimed is:
 1. An electronic device comprising: a camera;a display; a memory storing one or more instructions; and a processorconfigured to execute the one or more instructions to: obtain an imagecomprising at least one object through the camera, identify an objecttype of the at least one object, obtain spatial informationcorresponding to the identified object type, based on the object type,and control the display, to display a virtual object, based on theobtained spatial information.
 2. The electronic device of claim 1,wherein the processor is further configured to execute the one or moreinstructions to: transmit, to an external server, a spatial informationrequest signal comprising information about the object type, andreceive, from the external server, the spatial information correspondingto the object type.
 3. The electronic device of claim 1, wherein theprocessor is further configured to execute the one or more instructionsto: generate a three-dimensional (3D) shape information corresponding toa size and orientation of the object included in the image, based on thespatial information corresponding to the object type, and control thedisplay, to display, the virtual object, based on the 3D shapeinformation.
 4. The electronic device of claim 1, wherein the processoris further configured to execute the one or more instructions to controlthe display, to display, the virtual object such that the virtual objectappears around the at least one object, without displaying an area ofthe virtual object, obstructed by the at least one object.
 5. Theelectronic device of claim 1, wherein the processor is furtherconfigured to execute the one or more instructions to identify theobject type of the object included in the image by using a trained modeltrained by using an artificial intelligence model to identify an objecttype of an object.
 6. The electronic device of claim 1, wherein thespatial information corresponding to the object type includes a 3D shaperepresenting the object type.
 7. An operating method of an electronicdevice, the method comprising: obtaining an image comprising at leastone object through the camera; identifying an object type of the atleast one object; obtaining spatial information corresponding to theidentified object type, based on the object type; and controlling adisplay, to display, a virtual object, based on the obtained spatialinformation.
 8. The operating method of claim 7, wherein the obtainingof the spatial information comprises: transmitting a spatial informationrequest signal comprising information about the object type to anexternal server; and receiving the spatial information corresponding tothe object type from the external server.
 9. The operating method ofclaim 7, further comprising: generating a three-dimensional (3D) shapeinformation corresponding to a size and orientation of the objectincluded in the image, based on the spatial information corresponding tothe object type; and controlling the display, to display, the virtualobject, based on the 3D shape information.
 10. The operating method ofclaim 7, wherein the displaying of the virtual object comprisescontrolling the display, to display, the virtual object such that thevirtual object appears around the at least one object, withoutdisplaying an area of the virtual object, obstructed by the at least oneobject.
 11. The operating method of claim 7, wherein the identifying ofthe object type comprises identifying the object type of the objectincluded in the image, by using a trained model trained by using anartificial intelligence model to identify an object type of an object.12. The operating method of claim 7, wherein the spatial informationcorresponding to the object type includes a 3D shape representing theobject type.
 13. A non-transitory computer-readable recording mediumhaving recorded thereon a program for executing a method comprising:obtaining an image comprising at least one object through a camera;identifying an object type of the at least one object; obtaining spatialinformation corresponding to the identified object type; and controllinga display, to display, a virtual object, based on the obtained spatialinformation.